This invention relates to computer programming, more particularly, to a method and apparatus for controlling the timing of the invocation of events within a computer runtime environment, and specifically, timing the invocation of multimedia events within a Web page runtime environment.
The Internet and the World Wide Web are rapidly expanding, with businesses and individuals hosting their own Web pages. This has created a demand for richer Web page capabilities especially in the area of coordinated presentation and control of multimedia events, including being able to easily synchronize the execution of a plurality of multimedia events from a single point in time (i.e., playing a multimedia timeline). Because not all Web page owners are sophisticated computer users, the design and programming of Web pages must remain simple. Nor should synchronizing the execution of multimedia events to a timeline defined within in a Web page require complicated or lengthy user programs, or require the use of an external application which increases the cost of a solution. Instead designing and programming a Web page should be intuitive (i.e., xe2x80x9cuser-friendlyxe2x80x9d), while still providing sophisticated capabilities, such as prioritization of events which are scheduled for the same time and defining a window for execution of a time critical event.
Web pages are composed of various multimedia elements, controls, and applets as defined by the Hypertext Markup Language (HTML) for a given Web page. Multimedia can be characterized as some combination of visual media, audio media and time. Multimedia is an open environment, with timing being the common thread across all multimedia events. Multimedia experiences, such as the movement of graphics and the playing of sound, require coordinated execution of these events. For instance, the playing of a sound bite of a door slamming should be coordinated with the animation of the door closing so the sound is heard upon the visual closing of the door (and not before or after such time).
Providing synchronized multimedia experiences is complicated because timing control information is not inherent in the content of an HTML document. Past attempts at providing such synchronization and timing control of activities within a Web page have basically take on one of two forms: (1) external programs and (2) lengthy, complicated scripts or programs. These solutions generally are non-user-friendly, require additional expenses, and do not provide true synchronization of multiple events to a single timeline. Additionally, other approaches have not allowed synchronization between application-specific components (e.g., graphics and audio) and host-intrinsic components (e.g., HTML elements).
External multimedia control programs, such as Director, by Macromedia, can be expensive, and do not allow the direct creation and modification of a multimedia timeline by editing the HTML code. Rather, any changes and additions to the multimedia timeline must be made using the external program itself. Furthermore, the timing mechanism of some of these external programs are based on xe2x80x9cframesxe2x80x9d of time rather than directly on a time scale. A frame corresponds to a duration of time during which a set of defined activities are to be performed. Frames provide a method to sequentially perform sets of activities where there is some timing relationship based on frame rates and the time required for the sets of activities within the frame to be performed. However, individual events are not specified to be executed at a particular time (e.g., at time t=10.000 seconds), rather to execute within a frame (e.g., in frame 2) which does not provide an intuitive indication of the exact time for executing the event.
A second approach has been to write lengthy code or scripts within a control loop of the computer program which evaluates the current time each time through the control loop. In which case, the control loop can tie up an entire processing thread which does not allow other applications or processes to proceed, or at least requires extra computer cycles. Every time the control loop is executed, a large number of statements must be evaluated to determine whether to execute the actual events. This approach is less than desirable, especially as the number of events to control increases beyond one or two. In this case, scripts become long and complicated, and often more complex than a typical Web author knows how to create by hand. The control loop requires a large amount of processing time to determine whether to execute an event, which is disadvantageous especially in the multimedia context where events must be synchronized to a timeline having a granularity on the order of milliseconds. Furthermore, to guarantee a desired performance, the number of events controlled by a control loop must remain small because the control loop must iterate at a rate at least as great as the fastest timing requirement. Moreover, in many implementations such as those using an HTML document, the code within the control loop is written as an interpreted script thus compounding its evaluation time in comparison with that which is implemented using a built-in service or compiled code. Therefore, as the number of events to control becomes large, the timing performance correspondingly degrades and the granularity to which events can be set to perform becomes coarser. Moreover, the timing performance varies significantly and unpredictably depending on the power, performance, and current operating load of the computer on which these scripts or sets of code are run.
A variant of this approach uses a software timer control to generate a control event on a regular basis and causing a control program to execute. The control program then evaluates the current time and initiates the execution of the desired multimedia event. For instance, on a Web page, the setTimeout command can be used. This HTML code variant is more desirable than that of a control loop as it decreases the amount of evaluation code required. However, using this approach, it is cumbersome at best to stop the execution of sets of active timelines, and, more importantly, there is no synchronization of two events to a single timeline. Instead, these events are independently scheduled when their respective setTimeout command is interpreted and therefore, they are not synchronized. Furthermore, when multiple events are scheduled to begin execution at the same instant of time, the order in which the events will be executed is undefined for the setTimeout command (and commonly in other timing control approaches) which means a Web page does not necessarily produce the same results every time it is run, either on the same or on a different platform. Furthermore, this approach has disadvantageous performance limitations because the scripts become long and complex similar to that of an approach previously described herein, and the timing performance is unpredictably sensitive to the performance of the machine on which these set of code or scripts are executed.
In addition to their performance limitations, the prior approaches are directed towards basic timing control. In other words, these approaches only provide a way of initiating the execution of an activity at some defined point in the future, which is only a subset of the identified need in the computer industry for controlling multimedia experiences and applications. Lacking from these prior approaches are robust capabilities for providing true prioritization among events scheduled to happen at the same time, and a drop threshold capability to ensure that time sensitive events whose execution windows have expired are not executed.
In non-Web environments, schedulers have been provided to be able to start a process at a specified future time (e.g., a UNIX xe2x80x9cchronxe2x80x9d job). However, these schedulers do not provide the capabilities required for synchronizing a plurality of multimedia events along a timeline. For instance, the timing granularity of such schedulers is usually on the order of seconds, with no exact guaranteed starting time. Moreover, these schedulers do not accommodate single or multiple timelines with an accuracy of milliseconds for initiating actions as required for presentation of multimedia experiences. Moreover, they do not provide enhanced features for controlling the invocation of events such as drop threshold or priority arbitration mechanisms.
According to the invention, a sequencer control invokes events from an extensible set of events at their respective predetermined time(s), uses an arbitration mechanism to determine the execution order of multiple events scheduled for the same time, and provides a thresholding mechanism to determine whether not to invoke a scheduled event when an event""s actual invocation time exceeds its scheduled invocation time by its specified drop threshold time period. This sequencer control performs predictably and consistently on a wide range of client hardware configurations representing a broad range of performance profiles. Using this sequencer control to initiate multimedia events within a Web page, a deterministic presentation is provided whereby the multimedia events are synchronized according to a specification which runs the same on every browser hosting the sequencer control. Outside the Web page environment, a sequencer control provides a way of synchronizing the execution of events for a variety of applications and environments including operating and windowing systems. The terms xe2x80x9ceventxe2x80x9d and xe2x80x9cactionxe2x80x9d are interchangeably used herein to refer to any item that can be invoked by the sequencer control. The set of events which can be invoked by the sequencer control is extensible to encompass most any computer activity, including scripts, applications, objects, procedures, methods, interfaces, sending messages, generating a computer event, or the like.
More specifically, the invocation time for each multimedia event within the one or more extensible sequence sets of events is defined by a specification in an HTML document which can be created using a standard text editor. Each of these extensible sets of events corresponds to a single multimedia timeline. This specification includes the event, the time to invoke this event, and a sequence set. Additionally, this specification optionally includes (1) a loop count for the number of times to invoke the event, (2) an interval value quantifying the time between the invoking of repeating events, (3) a tie break number specifying a priority for determining an order of events when multiple events are scheduled for the same time, and (4) a drop threshold value identifying a window of time within which a scheduled event must be invoked or else it is ignored.
In keeping with the invention, the sequencer control accommodates variances in the starting times of events caused by processing delays of simultaneously scheduled events. When multiple events are scheduled for the same time, the invoking of some of these events will be delayed because only one event can be invoked at the same instant in time. This delay will vary according to the duration of the other events. As the sequencer control can accommodate an arbitrary number of sets of action with the actions being invoked at arbitrary time intervals, the order and timing of the invocation of events is not know a priori. Rather, it is only known at run time.
In one embodiment, the time required to invoke an event is determined by whether (1) the event is to execute in a thread apart from the sequencer control in which case the amount of delay is the time required to launch or interact with the event""s process, or (2) the event executes within the same thread as that of the sequencer control in which case the amount of delay is the time required for the event to run to completion. In another embodiment, certain events executing in a thread apart from the sequencer control are completed before events scheduled for a subsequent time or those with a higher tiebreak value are executed. For instance, these certain events include events having non-default tiebreak values. Techniques known to one of ordinary skill in the art are used to synchronize the execution of events across multiple threads and processes, whether executing on the same or different processors or computers.
To accommodate these delays, the sequencer control provides the two enhanced capabilities of a tie break arbitration mechanism and drop threshold. For each event, a tie break priority value can be specified, which defines a priority for invoking an event when multiple events are scheduled to be invoked at the same time. The invoking order is then determined by sorting the events based on their priority values. By exploiting the tie break number arbitration mechanism, a user can specify the sequence for invoking multiple events scheduled at the same time, rather than this ordering being undefined and varying between browser or sequencer control implementations.
In addition, the drop threshold value in the specification defines a threshold time in which an event can be invoked. If the invocation time of an event exceeds its predetermined drop threshold period, the event is not invoked.
When a Web page having sequencer control commands is loaded, a data structure for each of the sequence sets is populated with the event invoking information and parameters. The sequence set is then executed by an explicit play command, or alternatively, the sequence set could be executed automatically upon completion of loading the sequence set or Web page. The sequencer control then determines the first time at which it has a scheduled event, and passes a request to a clock object, which calls back the sequencer control at the appropriate time.
Upon callback, the sequencer control retrieves all the events scheduled to be invoked since the last callback, or at or prior to the current time if this is the first callback. These events are placed in an execution queue in order based first on the time the actions are due to execute, and secondarily on their tie break numbers. The order within the execution queue determines their invocation order of the events. Before invoking an event, the sequencer control evaluates the event""s drop threshold value, if specified, to ensure that the current time is within its invoking window, and if so, the event is invoked. Next, if the event is specified to repeat, it is re-scheduled for its next specified invoking time. After all the events in the queue have been invoked, the sequencer control determines the time for invoking the next event. If that time is in the future, then the sequencer control sets this time as the callback time with the clock routine, else the sequence controller queues and invokes these current events by following the above specified invoking procedure.
The sequencer control also provides real-time control of the playing of a timeline. The timeline can be played, paused, stopped, and the current execution time can be set to some time instant in the past or future using the seek command.